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Abstract 

This  paper  discusses  research  in  developing  DoD  acquisition  metrics  associated  with 
Systems  Engineering  activities  that  may  provide  greater  insight  into  the  technical 
performance  of  development  programs.  These  metrics  are  called  Systems  Engineering 
Applied  Leading  Indicators  (ALI).  We  examine  current  development  of  single  and  multi¬ 
factor  ALIs  that  have  been  developed  during  the  past  year  at  the  Naval  Air  Systems 
Command  (NAVAIR)  in  Patuxent  River,  MD.  The  development  methods,  early  examination 
of  ALI  utility,  and  user  acceptance  are  discussed.  The  authors  have  been  embedded  with 
the  NAVAIR  Systems  Engineering  Development  and  Implementation  Center  (SEDIC)  (the 
center  of  this  work  for  NAVAIR)  as  part  of  this  ALI  exploration. 
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Introduction  and  Problem  Definition 

Background 

What  is  the  role  of  systems  engineering  (SE)  in  the  acquisition  and  development  of 
systems?  The  professional  society  for  SE  (INCOSE)  defines  SE  as  follows: 

Systems  engineering  is  an  interdisciplinary  approach  and  means  to  enable  the 
realization  of  successful  systems.  It  focuses  on  defining  customer  needs  and 
required  functionality  early  in  the  development  cycle,  documenting  requirements,  and 
then  proceeding  with  design  synthesis  and  system  validation  while  considering  the 
complete  problem:  operations,  cost  and  schedule,  performance,  training  and  support, 
test,  manufacturing,  and  disposal.  SE  considers  both  the  business  and  the 
technical  needs  of  all  customers  with  the  goal  of  providing  a  quality  product 
that  meets  the  user  needs.  (INCOSE,  2010) 

The  principles,  practices,  and  methods  of  SE  are  well  defined  and  long  practiced  by 
Government  and  industry  (INCOSE,  2010;  NASA,  2007;  NAVY,  2004).  The  value  added  by 
disciplining  the  development  of  a  system  is  well  appreciated.  In  the  mid  1990s,  SE  practices 
were  augmented  with  the  concepts  of  SE  metrics  (INCOSE,  1995;  1998;  Roedler,  2005). 
Early  implementation  of  these  metrics  has  been  directed  at  the  measurement  of  the 
performance  of  the  SE  process  itself. 

In  the  bolded  part  of  the  definition  above,  SE  continues  to  expand  its  benefits  to 
include  not  only  the  development  team  but  also  all  customers  and  stakeholders  who  are 
maximally  interested  in  a  project/program  that  is  delivered  satisfying  cost,  schedule,  as  well 
as  technical  goals.  There  is  now  interest  in  the  SE  community  on  how  to  expand,  define, 
and  derive  metrics  and  methods  that  would  provide  predictive  or  prognostic  indicators  of  the 
success  of  a  development  effort  as  a  whole  (see  Error!  Reference  source  not  found.) 
(Rhodes,  Valerdi  &  Roedler,  2009).  While  the  existing  SE  metrics  and  methods  have 
typically  produced  lagging  and  inferred  indicators  of  the  health  and  status  of  a  development 
effort,  current  efforts  and  research  are  now  underway  to  examine  how  to  provide  direct 
leading  indicators,  derived  from  SE  and  applied  to  understanding  and  predicting  the 
technical  trajectory  of  the  aggregate  development  effort.  Because  we  are  applying  and 
focusing  the  concepts  of  SE  leading  indicators  (Roedler  &  Rhodes,  2007),  we  will  refer  to 
this  concept  as  SE  Applied  Leading  Indicators  (ALI)  for  the  remainder  of  this  paper. 
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Government/Industry  Partnership  Exploring  SE  Leading  Indicator 
Concepts  and  Application 

(Rhodes  et  al.,  2009) 


The  authors  set  out  attempting  to  focus  on  why  programs  fail  to  meet  user 
expectations  at  delivery.  Our  goals  were  to  determine  what  engineering  metrics  could  be 
defined  and  analyzed  to  provide  such  insight  where  programs  are  apparently  not  getting 
such  insight  today  (based  upon  failure  rates  of  system  qualification  testing  results).  This 
goal  lead  us  to  intersect  ongoing  efforts  related  to  SE  ALIs  that  we  determined  would 
provide  an  understanding  of  closely  related  metrics  and  processes  that  we  would  underpin 
our  investigation.  The  authors  have  been  supporting  and  co-researching  with  Naval  Air 
Systems  Command  (NAVAIR)  in  Patuxent  River,  MD,  to  examine  the  identification, 
relevance,  and  application  of  SE  ALIs.  NAVAIR  has  been  examining  the  ALI  concept 
through  engagement  with  acquisition  offices,  data  gathering  and  analysis,  formulation  of 
predictor  algorithms,  and  prototype  ALI  tool  development.  The  Systems  Engineering 
Development  and  Implementation  Center  (SEDIC)  is  conducting  this  NAVAIR  effort  in 
collaboration  with  working  groups  depicted  in  Figure  1 . 


Problem  Definition 

Program  managers  apply  well-proven  and  refined  program  metrics  and  control 
mechanisms  largely  based  upon  Earned  Value  Methods  (EVM).  The  EVM  fulcrum  metrics 
are  cost  and  schedule  that  generate  analysis  outputs  depicting  variances  from  plans  and 
estimates.  From  EVM  analysis,  program  cost  and  schedule  status  can  be  assessed  and 
projection  of  those  parameters  can  be  inferred.  Program  managers,  however,  are  not 
provided  abundant  metrics  that  can  provide  insights  into  the  technical  health  of  a 
development  effort  and  indications  of  the  trajectory  of  program  health,  good  or  bad.  Risk 
metrics  and  processes  provide  some  indications  of  technical  health  but  are  often  qualitative 
and  provide  little  algorithmic  opportunities  for  prognostics.  In  general,  program  managers 
are  faced  with  the  development  of  complex  systems,  use  EVM  and  risk  management 
effectively;  however,  programs  are  failing  to  fully  control  costs  and  can  routinely  exceed  cost 
estimates  by  25%  or  more  (see  Figure  2). 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


671 


Figure  S.1 

Distribution  of  Total  Cost  Growth  from  MS  II  Adjusted  for  Procurement  Quantity  Changes 


Figure  1.  Control  of  Cost  Growth  of  Programs  Remains  a  Challenge 

(Arena,  Robert,  Murray  &  Younossi,  2006) 

In  addition  to  the  quantity  of  programs  that  exceed  cost  estimates,  it  appears  that 
acquisition  cost  growth  can  be  attributed  to  causes  centered  upon  control  of  technical 
baselines  (see  Figure  3).  The  development  of  ALIs  is  intended  to  gain  much  more  granular 
insight  into  the  development  of  the  technical  baselines  as  soon  as  possible  to  allow  for  both 
assessment  and  predicted  program  performance  so  mitigation  can  be  applied.  In  summary, 
the  specific  problem  and  research  response  follows: 

Problem — Program  managers  do  not  have  access  to  adequate  technical  metrics  in 
order  to  provide  high  fidelity  assessment  of  technical  health  of  a  complex  system 
development  program  and  quantitative  prediction  of  technical  performance. 

Research  Question — Can  SE  technical  metrics  be  identified,  quantified,  and 
methodically  applied  to  complex  system  developments  to  provide  technical  assessment  and 
leading  indications  of  technical  program  performance  and  ultimate  success? 

Research  Objectives: 

•  Identify  relevant  data  supporting  the  development  of  ALIs, 

•  Identify  leading  indicators  tailored  to  systems  engineering  effectiveness, 

•  Prototype  ALI  user  tools  to  measure  relevance,  acceptance,  and  obtain 
feedback,  and 

•  Identify  new,  revised,  or  derived  metrics  to  support  refined  ALI  methods. 
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DEVELOPMENT  COST  GROWTH 
BY  CATEGORY 


AVERAGE  RDT&E  COST  GROWTH  FROM 
ACQUISITION  PROGRAM  BASELINE 
(ACROSS  ALL  PROGRAMS) 

65% 


TECHNICAL  BASELINES 
59% 

•  WEIGHT  •  I  TEST  ARTICLES 

•  VOLUME  •  #  TEST  FLIGHTS 

•COMMONALITY  -SOFTWARE 

•OTHERS 


Figure  2.  Cost  Growth  Largely  Impacted  by  Control  of  Key  Attributes  of 

Technical  Baselines 
(NAVAIR  4.2  Division  Analysis) 

Advance  Leading  Indicator  Concepts 

Technical  Measurements 

SE  processes  provide  metrics,  measurements,  and  analysis  activities  throughout 
systems  development.  These  technical  measurement  activities  provide  lead  system 
engineers  and  project  managers  insight  into  project  technical  performance  and  associated 
risks.  These  metrics  are  most  often  associated  with  Measures  of  Effectiveness  (MOEs), 
Measures  of  Performance  (MOPs),  and  Technical  Performance  Measures  (TPMs)  with 
associated  Key  Performance  Parameters  (KPPs)  and  Key  System  Attributes  (KSAs).  These 
associations  and  metrics  are  qualified  through  continual  testing  and  often  manifest 
themselves  graphically  using  control  chart  methods  (see  Figure  4). 


Figure  3.  Technical  Measures  Associated  with  MOEs,  MOPs,  and  TPMs  Guide 

Testing  and  Achieving  Specifications 

(Roedler,  2005) 
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The  above  technical  measurement  processes  are  often  focused  on  assessing  the 
progress  of  the  system  in  meeting  specification  as  development  unfolds.  Although  the 
development  of  ALIs  seems  similar  to  these  practices,  the  intent  of  ALIs  is  provide  a  more 
holistic  and  prognostic  assessment  of  the  technical  aspects  of  the  project  by  integrating  both 
system  technical  metrics  as  well  as  systems  engineering-derived  process  metrics.  ALIs, 
although  substantiated  in  historical  performance  of  similar  projects,  are  highly  forward- 
looking  and  technical-rich  in  fidelity.  They  are  intended  to  inform  the  project  technical 
approach  and  be  fully  integrated  with  the  program  management  approach  (see  Figure  5). 


Figure  4.  ALIs  Provide  Metrics  Rooted  in  SE  Technical  Approach  and  Supports 

Program  Management  Approach 

The  development  and  use  of  ALIs  are  intended  to  augment  existing  program/project 
management  methods,  not  replace  them.  Although  influenced  by  many  similar  metrics  (e.g., 
cost,  schedule,  etc.),  ALIs  are  derived  from  system  attribute  and  system  engineering  metrics 
to  produce  technical  health  and  prognostics  that  enhance  the  program  manager’s  overall 
assessment  and  direction  of  the  project  (see  Figure  6).  They  enrich  the  existing  EVM- 
derived  assessment  to  provide  project  leadership  higher  fidelity  project  technical  status  and 
direction  that  enables  greater  decision  analysis  completeness. 
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Figure  5.  SE  Applied  Leading  Indicators  (ALI)  Augment  Program  Management 

ALI  Approach 

Data  Selection  and  Collection 

The  development  of  ALIs  has  progressed  through  multiple  phases.  The  initial 
phases  identified  metrics  that  were  not  being  integrated  into  technical  assessment  that  could 
be  derived  from  engineering  activities.  The  ALI  effort  is  careful  to  augment  program 
management  evaluation  methods,  not  duplicate  them.  The  identification  of  metrics  led  to 
the  following  list  of  relevant  technical  metrics  that  were  considered  readily  available  in 
program  data  repositories: 

•  Aircraft  empty  weight, 

•  Software  metrics, 

•  Architecture  metrics, 

•  Requirements  metrics, 

•  Technical  review  closure  burn  down  rates, 

•  Reliability,  Availability,  and  Maintainability  (RAM)  metrics, 

•  Technical  risk  metrics, 

•  Engineering  staffing  metrics, 

•  System  complexity,  and 

•  Technology  maturity. 

The  data  above  is  collected  to  form  a  historical  baseline  of  program  performance  of 
similar  or  related  programs  (later  ALI  phases  would  incorporate  current  program  data  to 
predict  future  performance).  The  focus  of  early  research  has  been  associated  with  the 
technical  metric  of  aircraft  weight  and  subsequent  modeling  impacts  of  aircraft  weight- 
growth  to  program  cost-growth  throughout  development.  This  data  was  collected  from 
historical  records  from  the  program  division  records.  The  data  was  also  “affinitized”  or 
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grouped  in  like-program  categories  to  maintain  relevance  of  analysis  results.  Examples  of 
these  groupings  included  aircraft  development  with  similar  plan  forms  (e.g.,  rotary,  fixed 
wing,  remotely  piloted,  etc.),  size  of  program  (ACAT  I,  II,  etc.),  and  mission  (fighter, 
transports,  etc.).  In  all,  approximately  1 1  programs  form  the  foundation  for  further  data 
analysis. 


ALI  Data  Analysis 


Armed  with  a  single  dimension  of  ALI  applicability  (weight-growth  versus  cost- 
growth),  analysis  has  been  conducted  to  assess  what  statistical  methods  can  be  applied  to 
this  data  with  confidence  and  with  assurance  that  the  methods  are  representative,  relevant, 
and  extensible  to  provide  useful  data  to  program  managers.  The  steps  for  the  early  “single¬ 
factor”  ALI  analysis  is  depicted  in  Figure  7  and  is  summarized  below. 


%  CWG 
%  BP 
%  BNTE 
Etc. 


Correlation 

Analysis 


Figure  6.  ALI  Data  Analysis  Transforms  SE  Historical  Metrics  to  Algorithmic,  Leading 
Assessment  of  Technical  Performance  and  Prototype  ALI  Tool  Development 


•  Test  for  impact  (correlation) — Of  several  program  performance  impacts 
related  to  aircraft  weight  throughout  development,  cost  growth  proved  to  be 
highly  correlated  and  validated  and  formed  the  basis  for  continued  analysis. 

•  Test  for  meaningful  relationships  (regression) — The  ability  of  weight 
growth  to  predict  cost  growth  was  examined  through  several  regression 
methods.  The  results  showed  significant  statistical  strength  of  using  weight- 
growth  as  a  cost-growth  predictor;  however,  the  data  must  be  segmented  into 
major  epochs  of  program  development  to  maximize  this  predictive  strength. 
The  epochs  were  separated  by  major  design  reviews  (e.g.,  PDR,  CDR,  First 
Fight,  etc.)  to  ensure  predictive  usefulness. 

•  Derive  algorithms  to  develop  prognostics — The  segmented  data  produced 
regression  formulations  with  confidence  and  fit  parameters  for  each  epoch 
and  program  types.  These  formulations  established  historical  performance 
baselines. 

•  Validate  models  against  all  programs — The  data  was  validated  against  the 
affinitized  groupings  to  ensure  that  the  regression  models  derived  above 
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provided  an  accurate  model  of  historical  performance  of  the  programs  they 
modeled. 

•  Develop  “tripwire”  boundaries  prediction  zones — Although  a  prediction 
algorithm  was  considered  useful,  it  was  considered  most  valuable  if  cost- 
growth  boundaries  could  be  added  to  the  prediction.  These  boundaries  could 
be  adjusted  based  upon  the  interest  of  the  program  manager,  but  as  a 
minimum,  would  be  set  at  cost-growth  conditions  that  would  alert  the  program 
manager  and  leadership  of  severe  program  trouble  (e.g.,  10  USC  2433 
(Nunn-McCurdy)  limits).  These  limits  were  segmented  into  classic  color 
zones  (or  “tripwires”)  of  assessment  (e.g.,  red,  yellow,  and  green). 

•  Build  tool  for  user  validation — The  predictive  regression-based  formulation 
was  combined  with  the  tripwire  assessments/graphics  to  prototype  a  user  tool 
for  inputting  and  assessing  their  program  based  upon  current  weight 
estimates  and  comparing  their  current  weight  estimate  (at  a  particular  time  of 
the  development)  and  comparing  to  historical  performance  of  similar 
programs  to  provide  a  predictor  of  program  cost-growth  performan 

Early  Single-factor  ALI  Feedback 

The  user  community  (program  mangers  of  NAVAIR  acquisitions)  were  presented  the 
output  of  the  tool  is  depicted  in  Error!  Reference  source  not  found,  to  obtain  feedback 
related  to  ALI  concept  and  utility.  The  graphic  is  described  in  the  following  paragraph. 


Figure  7.  ALI  Prototype  Tool  Provides  Cost-growth  Prediction  Based  Upon  Similar  Program 

Histories  as  well  as  Status  of  Subject  Program 

(Denq,  Hein  &  Gilliard,  2009) 

The  starting  “dot”  in  the  yellow  zone  is  the  current  assessment  of  an  example 
program.  The  solid  black  line  is  the  predicted  cost  growth  of  the  program  based  upon  where 
the  current  program  is  in  comparison  to  historical  performance  of  similar  programs.  The 
dotted  lines  are  confidence  bounds  of  the  prediction.  This  graph,  once  again,  is  the 
depiction  of  the  impact  of  a  single  ALI  (weight  growth)  on  program  cost  growth.  Early 
program  manager  and  lead  engineer  feedback  revealed  that  this  graph,  although  informative 
to  a  degree,  generates  more  questions  than  it  answers.  Some  examples  include: 

If  single-factor  ALI  analysis  predicts  cost  growth,  what  other  factors  may  also 
impact  cost  growth? 
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•  What  are  the  impact  comparisons  among  single  ALIs? 

•  Do  other  ALIs  “mutual  couple”  to  cause  cost  growth? 

•  What  do  I  (PM/SE)  do  about  it? 

•  How  much  is  my  program  like  historical  programs? 

•  How  can  I  input  my  own  predictive  performance  judgment  into  the  algorithm? 

Moving  to  Multi-factor  ALIs 

The  most  generalized  feedback  from  program  managers  and  systems  engineers  to 
the  early  single-factor  ALI  concept  is  that  it  needs  (1)  to  consider  more  ALIs,  (2)  incorporate 
their  interactions,  and  (3)  algorithmically  combine  their  influences  into  an  integrated  ALI 
metric  for  the  program.  Similar  to  EVM  integration  of  cost,  schedule,  and  achievements 
(milestone  completion)  into  a  few  key  metrics,  ALI  needs  to  work  toward  that  goal.  The 
process  for  moving  to  an  integrated  ALI  output  is  shown  in  Figure  9. 


Figure  8.  Single-factor  ALI  Analyses  Are  First  Steps  to  an  Integrated  ALI  Output 

The  single-factor  ALI  analysis  and  formulations  are  shown  in  the  center  of  the 
diagram.  They  are  analyzed  individually  and  then,  after  model  validation,  are  integrated  to 
provide  a  more  “global”  ALI  metric.  The  repeated  analysis  steps  are  depicted  in  Figure  10. 
This  process  has  led  to  an  attempt  at  an  integrated,  multi-factor  ALI  approach  that  is 
currently  being  explored. 
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Figure  9.  Parallel  and  Independent  Single-factor  ALIs  Lead  to  an  Integrated  ALI  for  the 

Program 


Multi-factor  ALI  Development 

As  discussed  above,  single-factor  ALI  development  and  research  has  led  to  the 
current  research  into  multi-factor  ALIs.  The  underlying  assumption  is  that  if  a  single-factor 
ALI  concept  was  validated  historically,  proved  some  utility  in  prediction  program 
performance,  and  had  statistical  saliencies  that  could  be  exploited  in  a  tool,  then  we  may  be 
able  to  ingest  multiple  ALI  metrics  simultaneously  and  provide  meaningful  analysis  using 
related  statistical  methods  suited  for  multi-factor  analysis.  Ongoing  multi-factor  ALI 
investigation  (see  Figure  11): 

•  Retains  historical  data  analysis  of  key  program  ALI  metrics  (this  maintains  a 
credible  baseline  of  program  performance  upon  which  to  compare  programs). 

•  Applies  multiple  regression  methods. 

•  Integrates  user  assessment  of  both  current  conditions  and  their  predictions  of 
individual  ALI  future  performance  (e.g.,  if  your  program  is  currently  5%  over 
weight,  what  is  your  prediction  of  how  this  metric  will  change  in  the  future?). 

•  Applies  program  end-state  simulations  based  upon  historical  formulation  and 
user  estimates.  After  establishing  both  historical  baseline  and  associated 
multiple  regression  algorithmic  models,  user  predictions  are  integrated  into 
the  models  via  simulations  to  predict  program  performance,  fit,  and 
confidence  limits. 

•  Provides  integrated  multi-factor  ALI  graphical  output  to  the  program 
leadership. 
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Figure  10.  Multi-factor  ALI  Development/Research  Approach 


Early  graphical  concepts  are  intended  to  give  insights  into  the  “mutual  coupling” 
among  the  ALIs  and  their  impact  on  the  program.  Some  concepts  include  an  “interaction 
matrix”  approach  (Figure  12,  left)  showing,  for  example,  which  multiple  ALIs  drive  program 
cost  and  schedule  (indicated  by  colors)  and  provide  insight  into  their  possible  interactions 
(inferred  by  their  relationships  vertically  and  horizontally).  Additionally,  from  multi-factor  ALI 
analysis,  it  may  be  possible  to  depict  which  factors  are  most  influential  on  program 
performance 
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Figure  11.  Example  of  Multiple  Alis  Influencing  Program  (Left)  Cost  and  Schedule 
(Color)  and  Inferring  Their  Possible  Interactions  (Vertical/Horizontal 
Association)  and  (Right)  Key  ALI  Influencers 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


680 


ALI  Insight  into  System  Qualification  Testing  Success 

Consistent  with  the  authors’  original  goals,  an  NPS  capstone  project  thesis 
investigated  using  the  available  ALI  analysis  data  to  gain  insight  into  how  programs  were 
succeeding  in  their  qualification  testing  (Buchanan  &  Jungbluth,  2010).  Their  research 
indicates  some  promising,  although  weak,  statistical  inferences  about  the  data  and 
successful  testing  outcomes.  Their  work  sets  foundations  for  further  research  discussed 
below. 

Results  and  Conclusions 

Although  this  ALI  research  is  in  the  early  stages,  the  ALI  strategy,  methods,  and 
results  discussed  in  this  paper  show  promise  for  providing  program  manager  and  lead 
system  engineer  insight  into  the  current  and  predicted  technical  success  of  their  programs. 
This  has  been  demonstrated  through  ALI  data  analyses,  ALI  user  tool  prototypes,  and  user 
acceptance  testing. 

This  research  began  with  a  focus  on  why  programs  fail  to  meet  user  expectations  at 
delivery.  The  goal  is  to  determine  what  engineering  metrics  can  be  defined  and  analyzed  to 
provide  insight  into  success  of  qualification  testing  (e.g.,  operational  test  and  evaluation, 
validation,  etc.).  This  goal  lead  us  to  intersect  ongoing  efforts  related  to  SE  ALIs  that  we 
determined  would  provide  an  understanding  of  closely  related  metrics  and  processes  that 
would  underpin  our  investigation.  The  ALI  research  is  still  formative  and  evolving  and  the 
following  conclusions  are  mostly  qualitative  (non  parametric)  but  help  to  refine  further 
directions  related  to  ALIs  and  original  research  goals. 

Data — Although  there  are  rich  data  repositories  available  in  the  case  of  NAVAIR,  the 
data  can  be  inconsistent  and  incongruent.  This  increases  difficulty  in  data  analysis  and 
bounding  uncertainty  in  the  predictive  credibility  of  the  ALI  algorithms  and  tools. 

Additionally,  retention  of  data  from  various  programs  is  sometimes  incomplete,  leading  to 
statistical  analysis  of  sparse  data.  These  problems  are  not,  however,  insurmountable  and 
occur  regularly  in  statistical  analysis  activities.  The  benefit  of  the  ALI  investigation  is  that 
recommended  ALI  metrics  will  emerge  that  can  be  recommended  to  be  inculcated  into  the 
acquisitions  to  enable  greater  future  ALI  fidelity,  granularity,  and  reliability. 

Single-factor  ALI  analysis — The  weight-growth  versus  cost-growth  ALI  analysis 
revealed  that  the  development  method  was  valid,  provided  a  basis  for  ALI  tool  prototyping, 
and  garnered  preliminary  user  acceptance,  understanding,  suggested  improvements,  and 
identified  ALI  concept  shortfalls.  The  technical  basis  is  strong;  however,  the  most  impactful 
recommendation  from  users  was  to  demand  multi-factor  ALI  methods. 

Note:  When  we  tried  a  “programmatic”  metric  (staffing-growth  versus  cost-growth)  as 
a  comparison,  the  statistical  predictive  strength  was  not  as  strong  as  the  technical  metric  of 
weight.  The  resulting  conclusion  was  that  there  are  many  external  factors  (re-baselining, 
inter-program  staff  balancing,  etc.),  which  weakened  statistical  fit.  Additionally,  although  we 
have  some  interest  in  multi-ALI  interactions  with  programmatic  metrics,  we  discontinued  the 
staffing  investigation  because  it  proved  too  parallel  with  programmatic  metrics  (i.e.,  EVM). 

Multi-factor  analysis — These  methods  and  analyses  are  in  very  early  stages.  Early 
models  and  processes  are  employing  data  from  the  same  programs,  leveraging  single-factor 
analysis  lessons  learned,  expanding  to  include  multivariate  statistical  methods,  and  new 
graphical  output  techniques.  Early  indications  using  simulated  modeling  data  show  promise. 
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The  next  steps  will  include  actual  data,  validate  multivariate  models,  and  prototype  a  tool  to 
garner  user  acceptance. 

ALI  metric  expansion — The  only  metric  that  was  validated  was  aircraft  weight  and 
its  growth  throughout  the  development  cycle.  More  metrics  still  need  to  be  developed  and 
incorporated  into  the  research. 

User  acceptance — Users  recognize  the  need  for  a  method  based  upon  technical 
metrics  to  provide  predictive  program  performance  insight.  They  do  not,  however,  want  ALI 
to  replicate  EVM-based  metrics  and  methods.  Additionally,  they  desire  ALI  methods  to 
incorporate  prediction  inferences  and  judgments  of  the  project  engineering  and 
management  team  to  influence  analytical  output.  Finally,  as  stated  earlier,  user  inputs 
showed  a  strong  need  to  reveal  mutual  coupling  of  the  multiple  ALI  factors,  the  overall 
impact  to  the  program,  and  insights  into  how  to  respond,  technically. 

Areas  for  Continuing  Research 

Multi-factor  ALIs — As  stated  above,  this  analysis  is  in  the  early  phases  and  needs 
to  be  completed  to  the  point  of  testing,  validation,  and  user  acceptance/feedback.  The  next 
steps  are  to  include  ingesting  actual  data,  validating  multivariate  models,  and  prototyping  a 
tool/user  interface  to  gain  insight  into  user  acceptance 

Total-ownership-cost  control — During  the  conduct  of  this  research,  there  is  a  “sea 
change”  underway  toward  Total  Ownership  Cost  (TOC)  control  at  NAVAIR.  This  potentially 
shifts  the  types  of  ALI  metrics  but  the  fundamental  single  and  multi-factor  analysis  will,  most 
likely,  remain  viable.  The  nature  of  a  TOC  data  gathering,  algorithm  development,  and  tool 
may  have  to  be  reengineered  to  ensure  customer  acceptance  and  TOC  problem  relevance. 
Specifically,  the  following  areas  will  need  addressing: 

•  What  are  the  salient  TOC  assessment  goals  and  objectives? 

•  What  are  the  ALI  metrics  most  relevant  to  TOC  assessment? 

•  What  TOC  ALI  human  interaction  interfaces  would  be  most  useful  to  users? 

Qualification  and  acceptance  metrics — We  will  continue  to  investigate  how  ALI 
metrics  (or  derivatives)  might  be  viable  for  also  monitoring,  controlling,  predicting,  and 
maximizing  success  of  system  qualification  testing.  Expanding  and  defining  metrics  and 
methods  relative  to  predicting  and  analyzing  program  qualification  and  acceptance  test 
success  remains  a  goal. 
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Human  Resources 
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Logistics  Management 


Analysis  of  LAV  Depot  Maintenance 
Army  LOG  MOD 
ASDS  Product  Support  Analysis 
Cold-chain  Logistics 

Contractors  Supporting  Military  Operations 
Diffusion/Variability  on  Vendor  Performance  Evaluation 
Evolutionary  Acquisition 

Lean  Six  Sigma  to  Reduce  Costs  and  Improve  Readiness 
Naval  Aviation  Maintenance  and  Process  Improvement  (2) 

Optimizing  CIWS  Lifecycle  Support  (LCS) 
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■  Risk  Analysis  for  Performance-based  Logistics 

■  R-TOC  AEGIS  Microwave  Power  Tubes 

■  Sense-and-Respond  Logistics  Network 

■  Strategic  Sourcing 

Program  Management 

■  Building  Collaborative  Capacity 

■  Business  Process  Reengineering  (BPR)  for  LCS  Mission  Module  Acquisition 

■  Collaborative  IT  Tools  Leveraging  Competence 

■  Contractor  vs.  Organic  Support 

■  Knowledge,  Responsibilities  and  Decision  Rights  in  MDAPs 

■  KVA  Applied  to  AEGIS  and  SSDS 

■  Managing  the  Service  Supply  Chain 

■  Measuring  Uncertainty  in  Earned  Value 

■  Organizational  Modeling  and  Simulation 

■  Public-Private  Partnership 

■  Terminating  Your  Own  Program 

■  Utilizing  Collaborative  and  Three-dimensional  Imaging  Technology 

A  complete  listing  and  electronic  copies  of  published  research  are  available  on  our  website: 
www.acquisitionresearch.org 
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ALI  -  Applied  Leading  Indicators 
(Part  1  of  2) 


•  What  problem  are  we  trying  to  solve? 

•  How  do  ALIs  integrate  into  current 
management  practices? 

•  How  can  ALIs  benefit  programs? 
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What  is  the  Problem? 


Figure  5.1 

Distributer)  of  Total  Cost  Growth  from  MS  II Adjusted  for  Procurement  Quantity  Changes 
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The  SE  -  ALI  Challenge 


•  Can  we  provide  a  quantitative  projection  of  how 
varied  and  interrelated  technical  factors  are  impacting 
overall  program  performance? 

•  Can  we  provide  current  and  projected  program 
performance? 


•  Can  we  fill  a  gap  that  currently  exists  between 
technical  measures  and  overall  program  performance 
measures? 


•  Can  we  augment  current  program  health  and  status 
methods  and  tools  with  supporting  and  parallel 
technical  methods? 
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ALI  -  Augments  Program  Management  Processes 


•  Program  Metrics 

-  Cost  &  Schedule  -  based  (EVM) 

-  Focused  upon  actuals  vs.  planned  data 

-  Largely  measures  now  vs.  past 

-  Projecting  $/time-to-complete  based  upon  current  trajectories 

•  Risk  Management 

-  Cost,  schedule,  and  performance 

-  Assessments  heavily  based  on  history,  experience,  and  judgment 

-  Risk/issue  updates  are  based  on  now 

-  Root  cause  analysis  based  on  past  performance  and  helps  suggest  course  changes 

•  SE  Advanced  Leading  Indicators 

-  Collaborative  with  Program  and  Risk  metrics 

-  Future-focused  (prognostic) 

-  Performance/technical-focused  (vice  cost/schedule) 

-  Provides  needed  SE  insight  of  technical  interactions  and  dependencies  not  readily 
apparent  through  other  metrics 


ALI  Augments  Program  Management 
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Analysis  (EVMS) 
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SE  Leading  Indicators  Examples 


Requirements  volatility 

Design  definition  maturity  /  complexity 


Interface  maturity/  complexity 


Verification  &  validation  trends 
Technical  review  resolution  trends 
Technical  risks  trends 
Technology  maturity  &  adoption 
SE  staffing  &  skills 
SE  process  compliance 
NAVAIR-unique 

-  Aircraft  weight  trends 

-  ...(TBD) 
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ALI  -  Possible  Value-Added  Examples 


•  EVM  validation 

-  Your  program  looks  good  from  cost,  schedule,  and  milestone 
achievement.  How  does  your  program  technical  health 
compare  to  those  who  have  gone  before  you  with  similar 
EVM  assessments? 


*  Integrated  technical  assessment 

-  You  have  recently  re-baselined.  How  do  the  complexity  of 
your  design  and  recent  requirements  volatility  impact  your 
probability  of  meeting  program  and  performance  objectives? 


*  Risk  amplification 


-  Root  cause  analysis  suggest  several  course  corrections  for 
your  technical  approach.  What  SE  leading  indicators  help 
select  a  path? 
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ALI  -  Applied  Leading  Indicators 
(Part  2  of  2) 


•  How  are  we  building  ALIs? 

•  What  have  we  learned? 

•  Where  do  we  go  from  here? 
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SE  ALI  Process  Summary 


Cost  Schedule  Staffing 
Variance  Variance  Variance 


Aircraft  Weight 
Technology  Maturity 
System  Complexity 
Requirements  Volatility 
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SE  Performance 
Analysis 


Single  Factor  Leading  indicators 
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ALI  Analysis  Method 
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The  SE-ALI  Challenge  (Progressing  from  Single  to  Multiple 
Variable  Analysis) 


•  Program  performance 


-  Can  we  provide  current  and  projected 
program  performance  using  SE- based 
metrics?  (ALi  single  factor  process-  to 
date) 

Explaining  inter-relationships 

-  Can  we  provide  a  quantitative  projector 
of  how  varied  and  interrelated  technical 
factors  are  impacting  overall  program 
performance?  (emerging  Multi-  Variable 
process) 

Enhance  current  metrics 
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-  Can  we  fill  gaps  that  currently  exist  \  _ l  \  _ J 

between  technical  measures  and  overall  >  ' 

program  performance  measures?  ALI  single  Variable  ALI  Multi-Variable 

^  Analysis  Analysis 

•  Supporting  current  methods 


-  Can  we  augment  current  program  health 
and  status  methods  and  tools  with 
supporting  and  parallel  technical 
methods? 
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Multi-Factor  Technical  Approach 
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Conclusions  &  Lessons-Learned:  Data 


•  Data  can  be  inconsistent  and  incongruent. 

•  Retention  of  data  from  various  programs  is 
sometimes  incomplete  leading  to  statistical 
analysis  of  sparse  data. 


•  ALI  metrics  will  emerge  that  can  be 
recommended  to  be  inculcated  into  the 
acquisitions  to  enable  greater  future  ALI 
fidelity,  granularity,  and  reliability. 
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Conclusions  &  Lessons-Learned: 
Single  Factor  ALI 


•  Single  factor  ALI  analysis 

-  Development  method  was  valid 

-  Provided  a  basis  for  ALI  tool  prototyping 

-  Obtained  preliminary  user  acceptance, 
understanding,  suggested  improvements 

-  Identified  ALI  concept  shortfalls. 

-  Users  demand  multi-factor  ALI  methods 
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Conclusions  &  Lessons-Learned: 
Multi-Factor  Analysis 


•  In  very  early  stages. 

•  Leveraging  single-factor  analysis  lessons-learned 

•  Applying  multivariate  statistical  methods 

•  New  GUI  concepts 

•  Next  steps  will  expand  to  other  ALI  factors,  include 
actual  data,  validate  multivariate  models,  and 
prototype  a  tool  to  obtain  user  acceptance  feedback 
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Conclusions  &  Lessons-Learned:  User 
Acceptance 


•  Users  recognize  the  need  for  ALIs 

•  The  do  not,  however,  want  ALI  to  replicate  EVM- 
based  metrics  and  methods. 


•  They  desire  ALI  methods  to  incorporate  prediction 
inferences  and  judgments  of  the  project  engineering 
and  management  team  to  influence  analytical  output 


•  ALIs  need  to  reveal  mutual  coupling  of  the  multiple 
ALI  factors,  the  overall  impact  to  the  program,  and 
insights  into  how  to  respond,  technically. 
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Next  Steps 


•  Multi-factor  AL  Is 


•  “Sea  change”  underway  toward  Total 
Ownership  Cost  (TOC)  control  at  NAVAIR. 

-  What  are  the  salient  TOC  assessment  goals  and 
objectives? 

-  What  are  the  ALI  metrics  most  relevant  to  TOC 
assessment? 


-  What  TOC  ALI  human  interaction  interfaces  would 
be  most  useful  to  users? 
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